Blue ∗ : A Unified Programming Model for Diverse Data-intensive Cloud Computing Paradigms

Several computational paradigms exist today for processing large volumes of data on a cluster of resources: batch processing, iterative, interactive, memory-based, data flow oriented, relational, structured, etc. A unified system or framework that supports all paradigms would be greatly beneficial, as it will allow running different algorithms on the same dataset, reduce the system deployment, maintenance and training costs, provide a common platform for research and enable advances to be quickly incorporated. In this paper, we propose an approach towards such a system by introducing a unified programming model, called Blue. The model supports diverse paradigms of data-intensive computation, under the assumption that a program can be decomposed into a dependency graph of component tasks. In particular, Blue is capable of modeling iterative problems by a novel approach of unfolding cyclic graph into an unbounded acyclic graph. We illustrate the Blue model for several paradigms, such as Map-Reduce, joining datasets, Pregel, iterative algorithms such as k-Means and interactive querying. The model can be efficiently implemented by analyzing the graph and scheduling resources in manner that benefit from data and network locality. Additionally, Blue supports in-memory caching of data, explicitly by programmer or opportunistically by system, which has been shown to greatly improve the latency and throughput of interactive and iterative programs [1]. Finally, the Blue model provides simple and consistent semantics for fault-tolerance of acyclic as well as cyclic dependency graphs.